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Abstract Triality theory is proved for a general unconstrained global optimiza- 
tion problem. The method adopted is simple but mathematically rigorous. Results 
show that if the primal problem and its canonical dual have the same dimension, 
the triality theory holds strongly in the tri-duality form as it was originally pro- 
posed. Otherwise, both the canonical min-max duality and the double-max duality 
still hold strongly, but the double-min duality holds weakly in a super-symmetrical 
form as it was expected. Additionally, a complementary weak saddle min-max du- 
ality theorem is discovered. Therefore, an open problem on this statement left in 
2003 is solved completely. This theory can be used to identify not only the global 
minimum, but also the largest local minimum, maximum, and saddle points. Ap- 
plication is illustrated. Some fundamental concepts in optimization and remaining 
challenging problems in canonical duality theory are discussed. 

Key Words: Canonical duality, triality theory, Lagrangian, objectivity, canonical 
systems, global optimization. 

1 Introduction 

The general global optimization problem to be solved is proposed in the following 
form 

{V): e^l\^n{x) = W{x)+'^{x,Ax)-{xJ)\xeW'Y W 

where W{x) is a nonconvex function, A £ R"^" is a given symmetric matrix, 
/ G M" is a given vector (source), (*,*) is an inner product in M", and the 
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notation ext{*} stands for finding global extrema of the function given in {*}, 
including both global minimum and the largest local minimum and maximum. In 
order to have this general problem making sense in reality, the nonconvex function 
W{x) should obey certain fundamental rules in systems theory. 

Objectivity is a basic concept in science, which is often attributed with the 
property of scientific measurements that can be measured independently of the 
observer. General description of the objectivity can be easily found on internet and 
in many mathematical physics textbooks (see [271132] ). Mathematical definitions 
of the objective set and objective function are given in the book [9 (Chapter 6, 
page 288). Let 

Q = {Qe M"^^'"! = Q-\ detg = 1} 
be a proper orthogonal rotation group. 

Definition 1 (Objectivity and Isotropy) A subset C M.""^ is said to be 
objective if Qy G 3^a Vy G 3^a and VQ G Q. A real-valued function T : — > K 
is said to be objective if its domain is objective and 

T{Qy) = T{y) \fy e ya and VQ G Q. (2) 

A subset ya C K"" is said to be isotropic if yQ^ e ya Vy G 3^q and VQ £ Q- 
A real-valued function T : 3^a — > K is said to be isotropic if its domain is isotropic 
and 

TiyQ^) = T{y) Vy G J^a and VQ G Q. (3) 

Geometrically speaking, the objectivity means that the function T(y) does not 
depend on rotation, but on certain measure (norm) of its variable y. Therefore, 
the most simple objective function is the Z2-norm T(y) = ||y|| since ||Qy||^ = 
y^ Qy = y^y = \\v\\^ VQ G Q. While the isotropy implies that the function 
T(y) possesses a certain symmetry. By the fact that {xQ'^){xQ'^)^ = xx^ >^ 
VQ £ Q, the concept of isotropy plays important role in Semi-Definite Pro- 
gramming (SDP) and integer programming [131'18'. 

The objectivity in science is also refereed as frame invariance, which lays a 
foundation for mathematical physics and systems theory. In fact, the canonical 
duality theory was originally developed from this concept [9] , which is the reason 
why this theory can be applied not only for modeling and analysis of complex sys- 
tems, but also for solving a large class of nonconvex/nonsmooth/discrete problems 
in both mathematical physics and global optimization. In this paper, we shall need 
only the following weak assumptions for the nonconvex function W{x). 

(Al). The nonconvex function W{x) is twice continuously differentiable. 
(A2). There exits a geometrical operator 

A{x) = I ^x^B'^x + blx^ : R" ^ R'" (4) 

and a strictly convex function V : ISJ" R such that 



W{x) = V{A{x)), 
where B'' G R"""" and 6fc G R", fc = 1, • • • , m. 



(5) 
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(A3). The critical points of problem {V) are non-singular, i.e, if \7n{x) = 0, 
then det(V^77(S)) ^ 0. 

Based on Assumption (A2), the general problem ^ can be reformulated in 
the following canonical form: 



This problem arises extensively in many fields of engineering and sciences, includ- 
ing Euclidean distance geometry [4, 19 , computational biology [5'' 2 SI'S], numerical 
methods for solving a large class of nonconvex variational problems in mathemat- 
ical physics |12ll26l[3l| . and much more. 

Actually, the assumption (A2) is the so-called canonical transformation intro- 
duced in The idea of this transformation was from Gao and Strang's original 
work [21_ on nonconvex variational problems in large deformation theory, where 
the geometrical operator A{u) = ^(Vit)"^(Vu) is a Cauchy-Riemann metric ten- 
sor field, which is an objective measure of the deformation gradient e = Vu, and 
W{\7u) = V{A{u)) is a stored strain energy. By using finite element discretization 
for the deformation field u{x), the nonconvex variational problems in infinite di- 
mensional space can be reduced to the canonical global optimization problem {V) 
(see '26,3T). It is known in continuum physics that the stored energy W is usually 
a nonconvex function of the linear measure Vu (which is not a strain measure), 
but V{e) is convex in term of the objective measure e = A{u). Therefore, by this 
quadratic objective operator A{u), a complementary gap function was discovered 
by Gao and Strang in nonconvex variational analysis, and by which, complemen- 
tary variational principles were recovered in fully nonlinear equilibrium problems 
of mathematical physicsQ. They also proved that the nonnegative gap function 
can be used to identify global minimizer of the nonconvex problem. Seven years 
later, it was discovered that the negative gap function can be used to identify the 
largest local minimum and maximum. Therefore, the triality theory was first pro- 
posed in nonconvex mechanics [Sl[7], and then generalized to global optimization 
p.0 . This triality theory is composed of a canonical min-max duality and two pairs 
of double-min, double-max dualities, which reveals an intrinsic duality pattern in 
complex systems and has been used successfully for solving a wide class of chal- 
lenging problems in nonconvex analysis and global optimization [5^. However, it 
was realized in 2003 [ll.T2^ that the double-min duality holds conditionally under 
"certain additional conditions" . Recently, this problem is partly solved for a class 
of fourth order polynomial optimization problems |23l[36] . 

The aim of this paper is to prove the triality theory for the general nonconvex 
global optimization problem {V). In the following sections, we first provide a brief 
review on the canonical duality theory and the associated triality theory. We will 
show that by the canonical transformation, the nonconvex primal problem (V) 
can be reformulated as a canonical dual problem without duality gap. Section 
3 presents a strong triality theory for the case that the primal problem and its 

^ In continuum physics, complementary variational principle means perfect duality since any 
duality gap will violate certain physical laws. The existence of a complementary variational 
principle was a well-known debate existing for several decades in large deformation theory (see 
[30]). This problem was partially solved by Gao and Strang's work, and solved completely in 




(6) 



1999 [8]. 
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canonical dual have the same dimension, i.e. n = m. We then show in Section 4 that 
this theory holds weakly for the case n ^ m. The "certain additional conditions" 
for the double-min duality are provided. Application is illustrated in Section 5. 
The original definition of Lagrangian, Lagrangian duality and its difference with 
the canonical duality are discussed in Section 6. The paper is ended with some 
conclusion remarks and challenging problems. 

2 Canonical Duality, Triality, and Open Problem 

Let 

Va = {^e M""! I = A{x) \fx e M"}, 

By (Al) and (A2) we know that V : Va — > M is also a twice continuously differ- 
entiable. Therefore, its Legendre conjugate V* : — > M can be uniquely defined 

as 

y*(0 = sta{($;^>-y(0 Va}, (7) 

where (*;*) is an inner product in M'" and sta{ } stands for finding stationary 
value of the expression given in { }. ft is easy to verify that the canonical duality 
relations 

^ = Vy(0 ^ $ = VV*{c;) ^ V{i) + V*{<;) = {$;^) (8) 
hold on VaXV*. 

Substituting V{A{x)) = {A{x);<;) — V*{';), the primal function n{x) can be 
reformulated as the total-complementary function [9] 

S{x,<i) = ^{x,G{^)x} - - (x,F{<;)), (9) 

where 

m m 

G(<;) = A + ^ F{<;) = f -J^^kbk- 

fc=i fc=i 

For a fixed the criticality condition 'Vx^{x, <;) = leads to the following canon- 
ical equilibrium equation 

G{^)x = F{^), (10) 

which can be solved analytically to obtairQ x = [G{<;)]~^F{(;) for all <; in the 
canonical dual feasible space Sa defined by 

Sa = {^ev:\ F{^)eCoi (G(0)}, 

where Coi {G {<;)) is a space spanned by the columns of G{<;). Therefore, substi- 
tuting this solution into the total complementary function H, the canonical dual 
problem can be formulated as 

iV"): ext|77'^(<;) = -i([G(<;)]-ii^(0,FW>-V*(<j) l^e^.j. (11) 

The following theorem was originally presented in general nonconvex systems [9] . 
^ In this paper should be understood as a generalized inverse if dot G = 1101 
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Theorem 1 (Analytical Solution and Complementary-dual principle) 

Problem (V'^J is canonically dual to (V) in the sense that if ^ is a critical point of 
{V^), then 

x = [G{<;)]-'F{<;) (12) 
is a critical point of (V), the pair {x, ^) is a critical point of S{x, ^), and 

n{x) = s{x,-^) = n^{^). (13) 

This theorem shows that there is no duality gap between the primal problem 
{V) and its canonical dual {V^). Actually, in S{x, <i) the first term 

Gap[x,<i) = ^{x,G{<;)x) (14) 

is the complementary gap function, first introduced by Gao and Strang in 1989 
[21] . They proved that if this gap function is positive, the critical point ^ is a 
global maximizer of 77'^ and the associated x{^) is a global minimizer of the primal 
problem (V). By introducing the following notations 

St = {^eSa\G{<;)^G}, (15) 
S- ={^eSa\G{<;)^Q], (16) 

where G (^) >i means that G (<;) is positive semi-definite and G (<;)-< means 
that G {<;) is negative definite, the Gao and Strang canonical min-max duality 
theory can be stated as 

nix) = min nix) = max 77'^(<;) = n'^i^). (17) 

This general result has been used extensively in nonconvex analysis and mechanics 
[9ll43] ■ In 1996, it was discovered by Gao that if the gap function is negative in a 
neighborhood Xo y- So C R" x Sa of (x,^), then either the double- max duality 
relation 

n{x) = max n{x) = max77'^(<;) = 77'^(^) (18) 
holds or the double-min duality relation 

n{x) = min n{x) = min 77'*(^) = 77'*(^). (19) 

Therefore, the triality theorem was formed by these three pairs of dualities and 
has been used extensively in nonconvex mechanics 9,16 and global optimization 
[2l l20l[33] . However, it was realized in 2003 [111112] that if the dimensions of the 
primal problem and its canonical dual are different, the double-min duality (|19|) 
needs "certain additional conditions" . For the sake of mathematical rigor, the 
double-min duality was not included in the triality theory and these additional 
constraints were left as an open problem (see Remark 1 in [11], also Theorem 3 
and its Remark in a review article by Gao [12]). By the facts that the double-max 
duality (|18p is always true and the double-min duality plays a key role in real- 
life applications, it was still included in the triality theory in the either-or form 
in many applications for the purposes of perfection in esthesis and some other 
reasons in reality. In the following sections, we will show that the triality theorem 
holds strongly for the problems it was originally proposed. Also we will explain 
the reasons why the "certain additional conditions" in the double-min duality were 
ignored. 
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3 Strong Triality Theory 

In the case n = m, the triahty theorem holds strongly in the following form. 

Theorem 2 (Tri-duality Theorem) Suppose that ^ is a critical point of the 
canonical problem {V'^) and x = [G (^)]~^ -f^(^)- 

If ^ a Sa ^ then ^ is a global maximizer of Problem (V'^) in Sa if and only 
if X is a global minimizer of Problem iV), i.e., the following canonical min-max 
duality statement holds: 

n{x) = min 77 {x) <^ max 77^* (^) = 77'^(^). (20) 

If ^ £ Sa , then there exists a neighborhood Xo x So G M" x Sa of {x, ^) such 
that we have either the double-min duality statement 

n{x) = min 77 {x) <^ min 77^* (^) = 77'^ (^) , (21) 

or the double-max duality statement 

77(S) = max 77 (a;) <^ max77'' (?) = 77'^ (^) . (22) 

Proof. If (S,?) is a critical point of the total complementary function S'(a;,?), 
then by Theorem 1, we haves = [G(5)]"^F(^), and 

V'77'*(^) = -{S/A{x)f[G{^)]-^\/A{x) - VV*(^), (23) 

V^77(S) = G(^) + Vyl(S)vV(yl(S))(Vyl(S))^. (24) 
By the assumption (A2) we know that V{S,) is strictly convex, then, 

v2(y(vl(a^))) = (VV*(^))-V0, (25) 

where | = A{x). Substituting (|25p into (|24ll . we obtain 

V'77(S) = G(^) + V^(a^)(vV*(^))-^(V^(a;))^. (26) 

— Proof of the canonical min-max duality statement (|20p (this proof is a finite- 
dimensional version of Gao and Strangs original proof of Theorem 2 in non- 
convex analysis [21). 

Suppose that ^ £ Sa is a critical point. Since 77'^(?) is concave on Sa , the critical 
point ^ G Sa must be a global maximizer of 77'^(?) on Sa ■ 

On the other hand, if ? G Sa , the gap function Gap{x,^) = ^{x,G{^)x) is 
convex in a; G R". By the convexity of y : Va — > R, we have [21_ 

77(a;) - 77(a;) > (VV{A{x)); A{x) - A{x)) + ^(x,Ax) - ^(x,Ax} - (x - xj) 

= Gap{x, ^) - Gap{x, ^) - {X - X, F(^)) 

>{x-x, G{^)x - F{^)) = Vcc G M". 

Thus, X = [G(^)]~^7^(^) is a global minimizer of problem {V). Furthermore, ^ is 
also a global maximizer of Problem {V^) in Sa and the statement (|20[) holds by 
Theorem 1. 
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— Proof of the double-min duality statement (|2ip . 

Suppose that ^ £ Sa and ^ is a local minimizer of problem {T"'). Then, we have 
V'^n'^is) h and 



{VA{x)f[G{^)]-'\7A{x) h VV*(?) >- 0. 



Thus, VA{x) is invertible, which leads to 



G(^) ^ \7A{x){\/W*{^))-\\7A{x)f. 



(27) 



Therefore, we have 



\7^n{x) = G{^)+\7A{x){\/W*{^))-\\/A{x)f >- 0. 



(28) 



By the assumption (A3), x = [G(^)] ^-F(^) is also a local minimizer of problem 
{V). The reversed statement can be proved in the similar way. Thus, (|2ip holds. 

— Proof of the double-max duality statement (|22p . 

Suppose that ^ £ 5^ and ^ is a local maximizer of problem (V^). Then, V^77'^(^) ^ 
0. By Theorem 1, x = [G{^)]~^ F{^) is a critical point of problem (V). Due to the 
assumption (A3), V^77(a;) is invertible. By the well-known Sherman- Morrison- 
Woodbury identity V^77'^(^) is also invertible. Furthermore, 



(V"77(x))-^ = G{^y' +G{^)-'\7A{x){\/^n''{^))-\\/A{x)fG{^)-^ ^ 0. 



Thus, X = [G(<;)] (<f) is also a local maximizer of problem (V). Similarly, we 
can prove the reversed statement. Therefore, the triality theorem holds strongly 



Remark 1 The tri-duality theorem provides global extremum criteria for three 
types solutions of the nonconvex problem {V): a global minimizer x{i,) if ^ G Sa 
and a pair of the largest- valued local extrema, i.e., x(t,) is a global maximizer (resp. 
minimizer) if ^ G Sa is a local maximizer (reps, minimizer). This pair of largest 
local extrema plays a critical role in nonconvex mechanics and phase transitions. 

Remark 2 The tri-duality theorem can also be used to identify saddle points of 
the primal problem, i.e. ^ G Sa is a saddle point of 77(?) if and only if a; = 
\G{^)]~^ F{^) is a saddle point of n{x). By the facts that the saddle points are 
not stable and do not exist physically, these points are excluded from the triality 
theory. 

The triality theory was first discovered in post-buckling analysis of a large 
deformed elastic beam model proposed by Gao in 1996 [6l[7], where the primal 
functional is a double-well potential of a 2-dimensional displacement field, and 
its canonical dual is the so-called pure complementary energy defined on a 2- 
dimensional stress field. Therefore, the triality theory was first proposed in its 
strong form, i.e. the tri-duality theorem. 



for the case n = m. 



□ 
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4 Triality Theory for General Case 



We now consider the general case m ^ n. Suppose that x and ^ are the critical 
points of problem (V) and (V^), respectively, where x = [G{^)]~^F{^) and G(^) 
is invertible. In this case, we also can show that 



V^n{x) = G(^) + VA{x){V^V*{^))-\VA{x)f, 



and 



-{VA{x)f[Gm-WA{x) - VV*(^). 



(29) 



(30) 



Suppose that m < n. By the Sherman-Morison- Woodbury Theorem in [3 and 
the assumption (A3), we have 

[y^n{x)]-^ = [Gis)]-' + [G{^)]-'VA{x){V^n\^))-\\JA{x))^[G{^)]-\ (31) 



This shows that V 77 (<;) is invertible. Similarly, we can show that V 77 {<;) is 
also invertible if m > n. 

Lemma 1 Suppose that m < n, the critical point ^ G Sa is a local minimizer 
of problem (V"^). Then, \7^n{x) has m positive eigenvalues and n — m negative 
eigenvalues, i.e., there exists two matrices P\, £ ^j^j^ (z -^rix(n-m) ^^^^ 

that 

P^V^n{x)P\, > and P^V^n{x)Pf -< 0. (32) 

Proof. Since V^V"*(^) >- 0, there exists a invertible matrix 7? e R""^"" such that 
VV*(^) = 7?'^7i. Thus, we have 

- {VA{x)R-'f[G{^)]-'VA{x)R-' - 7^xm y 0. (33) 

Note that G{^) -< and VA{x)R~^{VA{x)R~^))'^ >z 0. There exists a matrix T 
such that 



T'G(^)T = Diag(-Ai, 



-An), 



(34) 



and 



T'^VA{x)R-\VA{x)R-^)fT = Biag{ai,--- ,arr^,0,■■■ ,0), 



(35) 

where Ajt > 0, A; = 1, • • • , n, and ak > 0,k = 1, • • • , m. According to the decom- 
position theory of singular matrices, we know that there exist orthogonal matrices 
U e M"""" and E e R'^x™ such that 



t'^VA{x)R~^ = U 







E. 



(36) 



V ••• y 

In light of (|35p . we know that U = Inxn- Then, we have 



{R-Yv^n'^{^)R-^ = -{VA{x)R-Y[G{^)r^VA{x)R-^ - I„ 



{T'^VA{x)R-^f[T^G{^)T]-^TVA{x)R-^ 

i.-.^-Ae^q. 
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Thus, ak > Xk, fc = 1, ■ ■ ■ , m. It is easy to verify that 

T^V'^n{x)T = Diag(ai - Ai, ■ ■ ■ , a™ - Am, -A^+i, ■ • • , -A„). (37) 

This shows that V^77(x) has m positive eigenvalues and n — m negative eigenval- 
ues. Therefore, the matrix P\, can be obtained by collecting all the eigenvectors 
corresponding to the positive eigenvalues and can be obtained by collecting all 
the eigenvectors corresponding to the negative eigenvalues. □ 
In a similar way, we can prove the following lemma. 

Lemma 2 Suppose that m > n and the critical point x = [G {^)]~^ F{^) is a 
local minimizer of Problem {V), where ^ G Sa ■ Then, V^i7'^(^) has n positive 
eigenvalues and m — n negative eigenvalues, i.e., there exists two matrices Q\, £ 
K""^" and G Kmx(m-«) ^^^f^ ^^^^^ 

QrV^J7''(^)Qb >- and Q[V^7T'*(^)Qtt ^ 0. (38) 

Let the m column vectors of Pi, be p^^ , • • • , and the n column vectors of 
Q\, he q\, - ■ ■ ,ql^, respectively. Clearly, p\, - ■ ■ ,p^ and q\, - ■ ■ , are two sets of 
linearly independent vectors, respectively. By introducing two subspaces 

Xi, = {x eKJ' \ x = x + eip\ + • • • + 6•„^p^, 6>^ G R, I = 1, • • • , m}, (39) 
<Sb = G M'" I ? = ^ + '&iq\ +■■■+ 'dnqn, i?i G M, i = 1, ■ ■ ■ , n}, (40) 

the triality theory holds for general case in the following refined form. 

Theorem 3 (Triality Theorem) 

Suppose that ^ is a critical point of problem (V^) and x = [G{^)]~^F{^). 
// ^ G Sa , then the canonical min-max duality holds in the strong form of 

n{x) = min n {x) ^ max TI'^ (^) = n'^{^). (41) 

If ^ £ Sa , then there exists a neighborhood Xo x So C K" x Sa of (x,^) such 
that the double-max duality holds in the strong form of 

iT(S) = maxJ7(x) OmaxiT''(?) =7T''(^). (42) 

However, the double-min duality statement holds conditionally in the following 
symmetrical forms. 

1. If m < n and ^ G Sa is a local minimizer of n'^{<;), then x = [G{^)]~^ F[^) is 
a saddle point of n{x) and the double-min duality holds weakly on XoC\X\, xSo, 
i.e. 

n{x) = min n (x) = min il'* {<;) = n'^{(.). (43) 

2. If m > n and x = [G(^)]~^F(^) is a local minimizer of II{x), then ^ is a 
saddle point of n'^{<;) and the double-min duality holds weakly on XoXSo(^S\,, 
i.e. 

n{x) = min 77 {x) = min 77'^ (?) = iT'*(t). (44) 
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Proof. The proof of min-max duality statement (|41(1 and the double-max duahty 
statement (|42p are the same to the proof of (|20p and (|22p . We only need to prove 
(1121) and im). 

Suppose that m < n and ^ G 5^^" is not only a local minimizer, but also a critical 
point of problem {V'^). By Lemma 1 we know that V^77(x) has both positive and 
negative eigenvalues. Thus, x = [G{^)]~^ F{^) is a saddle point of problem (V). 
We let 

<f{ti, ■■■ ,t^) = n{x + tip\ + ■■■ + tmP'^), (45) 

where p\^ - ■ ■ , are the column vectors of P\, defined in Lemma 1 . By direct 
verification, we have 

V^(0,--- ,0) = (V77(a;))^n=0 (46) 

and 

VV(0,--- ,0) = n^V"77(S)n^0. (47) 

Thus, (0, • • • ,0) is a local minimizer of ip{t\,--- ,tm)- Hence, the equation (|43p 
holds. The statement (|44ll can be proved in the similar way. □ 

Remark 3 (NP-hard Problems and Perturbation) The canonical min-max duality 
(|41|) shows that the nonconvex minimization problem is equivalent to a concave 
maximization dual problem over a closed convex set Sa ■ If n'^ (^ ) has at least one 
critical point in St , the global minimizer of 11 [x) can be easily obtained by the 
canonical duality theory. However, if n'^{<;) has no critical points in Sa , to find 
global minimizer for nonconvex function n{x) could be very difficult. If the vector 
/ = e M", the problem (P) is homogenous. Moreover, if bfe = e R" Vfc = 
1, • • • ,m, then the geometrical operator A{x) is a pure quadratic measure (i.e. 
an objective measure in certain space). In this case, the vector F{<;) = 0, the 
set Sa is empty, and the canonical dual function n'^{<;) = —V*{<;) is concave, 
which has only a unique maximizer ^. By the double-max duality we know that 
the corresponding primal solution x = is a local maximizer if ^ G Sa ■ From 
the point view of systems theory, the pure quadratic operator A{x) means that 
the system possesses certain symmetry. If there is no input (/ = 0), the primal 
function n{x) could have multiple global minimizers. It was indicated in [13] that 
a nonconvex minimization problem could be NP-hard if its canonical dual has 
no KKT (or critical) point in Sa ■ In order to solve this type problems, several 
perturbation methods have been suggested in |18ll33lll2] . It is shown very recently 
that by the canonical duality theory, a class of NP-hard box/integer constrained 
programming problems are equivalent to unconstrained canonical dual problems 
in continuous space, which can be solved via deterministic methods (22] . 

Dual to X[, and Si,, we can let 
A-)) = {a; e R" I a; = x + Oipl + • • • + 6l„-^p^_^, 61, e R, i = 1, • • • , n - m}, 
5o = e R"" I ? = ^ + Siql + ■■■+ -dr^-nqL-n, G R, I = 1 , • • • , m - Tl} , 

where {pf} and {qf} are column vectors of and Qj, respectively. Then, com- 
plementary to the weak double-min duality statements (|43|) and (|44|) , we have the 
following weak saddle min-max duality theorem. 
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Theorem 4 (Weak Saddle Duality Theorem) 

Suppose that t, £ Sa is a critical point of problem (V^ ), the vector x = 

is a neighborhood of (a;,^). 

1. If m < n and ^ 6 Sa is a local minimizer of n'^{<;), then x = [G{^)]~^ F{^) 
is a saddle point of n(x) and the saddle max-min duality holds weakly on 
XoC\X^X So, i-e. 

n{x) = max 77 {x) = min H'' {<;) = 77'*(^). (48) 

2. Ifm > n and x = [G{^)]~^ F{^) is a local minimizer of n{x), then ^ is a saddle 
point of n'^{<;) and the saddle min-max duality holds weakly on Xo x 5o n (Sj, 
i.e. 

n{x) = min 77 (a;) = max U"^ = H'^is). (49) 

Remark 4 Tiieorem 3 stiows tiiat botli tlie canonical min-max and double-max 
duality statements hold strongly for general cases; the double-min duality holds 
strongly for n = m but weakly for n ^ m in a symmetrical form. The "certain 
additional conditions" are simply the intersection Xo f] X\, for n > m and So H 
for n < m. Therefore, the open problem on the double-min duality left in 2003 [111 
[T2] is now solved! While from Theorem 4 we know that if G(^) ~< and n > m, the 
solution x{^) could be a saddle point. Mathematically speaking, nonstable critical 
points do not produce any computational difficulties in numerical optimization. 
Also, in real-life problems the saddle point is not considered as a phase state and 
does not physically exist. These are the part of reasons why the saddle point in 
Sa is ignored by the triality theory. 

The triality theory has been challenged recently by a large number of counter- 
examples in a series of more than seven papers (see [38[|40j and references cited 
therein). It was written in [38] that "Because our counter-examples are very sim- 
ple, using quadratic functions defined on whole Hilbert (even finite dimensional) 
spaces, it is difficult to reinforce the hypotheses of the above mentioned results in 
order to keep the same conclusions and not obtain trivialities." It turns out that 
in addition to many conceptual mistakes (see Section 6), most of these counter- 
examples simply discuss the saddle points in Sa for the case n ^ m. In fact, these 
count-examples address the same type of open problem for the double-min duality 
left unaddressed in [111112^1 . Indeed, by Theorem 3 we know that the double-min 
duality holds conditionally when n ^ m. Based on Theorems 2 and 3, we know 
that the saddle points could exist in Sa even if n = m; While by Theorem 4 one 
can easily construct many other V-Z type counter-examples which are physically 
useless. 



5 Application 

Let us consider the following quadratic-log optimization problem: 

{V): exti^n{x) = ^x^Ax-f2\og{^x'^B''x + d'')-x'^f\ a;eR"|, (50) 



^ It is interesting to note that the references I11II12| never been cited in any one of this set 
of papers. 
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where A is a positive definite matrix, B'' , k = 1, ■ ■ ■ ,m are positive semi-definite 
matrices and d*^ > 0, fc = 1, • • • , m. In this case, its canonical dual problem can be 
expressed as: 

(51) 

where 5a = {<; = {<^^} e R"' | - < ^fe < 0, /c = 1, • • • , m}. 
Let n = 2, m = 1, and 



2J ' \ AJ \0.l 

In this case, we have iSa = G M | — 1 < <; < 0} and 

n U) = 1 h ^ + 1 + logf-^) 




From Fig. 1 we can see that 7T'^(^) has one critical point <;i = —0.13696432 in 
5+ = (-0.2,0) and two critical points ^2 = -0.54470504 and ^3 = -0.95209751 
in Sa = (-1,-0.5). Thus, by the triality theorem we know that xi = {A + 
^iB^)~^f = [1.58640312,0.06886375]^ is the global minimizer to the problem 
{V). Since ^2 & Sa is a local minimizer, ^3 £ Sa is a local maximizer to the 
problem {V^), and m = 1 < n = 2, by Theorem 3 we know that X2 = {A + 
^2B^)"V = [-0.2901031,-0.5592211]'^ is a saddle point of n{x), while X3 = 
{A + q3B^)~^f = [-0.13296148,-0.0552978]^ is a local maximizer to the problem 
{V) ■ More applications can be found in [23] . 
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a b 

Fig. 2 Contours of 11 (x): (a) global minimizer of n(x) and (b) local maximizer of 11 (x) 

6 Some Fundamental Concepts in Canonical Systems 

Global optimization problem in mathematics is usually formulated in the following 
general form 

min{/(a;)| xeX C M"}, 

where the real- valued function f{x) is simply assumed to be nonconvex (or Lips- 
chitz, differentiable, etc.) on its feasible space X G M", in which certain constraints 
are given. It is known that this problem could have a large number of local extrema 
and to identify global optima is a main challenging task in global optimization. If 
there is no detailed information available for the given function f{x), it is difficult 
(may be impossible) to have a general theory and method for solving this gen- 
eral problem effectively. Also, to find the largest local extrema is fundamentally 
important in many real-life application^. 

Mathematics and physics (mechanics) have been complementary partners since 
Newton's time. It is known that the calculus of variation and mathematical op- 
timization were originally developed from Euler-Lagrange mechanics. Also, the 
modern mathematical theory of convex analysis was started from J.J. Moreau's 
pioneering work in contact mechanics [31]. However, as V.I. Arnold pointed out 
[1]: "In the middle of the twentieth century it was attempted to divide physics 
and mathematics. The consequences turned out to be catastrophic." For example, 
in mathematical physics, the objectivity is directly related to some fundamental 
concepts and principles, such as geometrical nonlinearity, constitutive laws, and 
work-conjugate principle, etc. A function(al) can be called objective or free energy 
only if certain intrinsic constraints (physical laws) are satisfied (see [16 ). Unfor- 
tunately, the objective function in mathematical optimization has been misused 
with other concepts such as cost function, energy function, and energy functionajf], 
which leads to some conceptual mistakes. This section will discuss some important 



* It should be emphasized here that to find the largest local maximum of f(x) is not simply 
equivalent to solve the problem min{— /(cc)| x £ X}. 
^ See the web page at |http: / /en.wikipedia.org/wiki/MathematicaLoptimization| 
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issues in classical Lagrangian mechanics/duality, mathematical optimization, and 
general systems theory. 



6.1 Canonical systems 

According to E. Tonti [39] , in virtually every physical system there exists at least 
three types of variables: 

(1) the configuration variable x X , which describes the state or output of 
the system, such as the Lagrangian generalized coordinates (or displacements) in 
analytical mechanics [25], decision variable in game theory, etc. 

(2) the source variable x* = f £ X* , which represents the input of the system, 
such as the external force in mechanics and charge density in theory of electrical 
field, etc. 

(3) a pair of internal (or intermediate) variables {£,£*) G £ x £* , which de- 
scribes certain interior (constitutive) properties of the system, such as strain and 
stress in elasticity, velocity and momentum in dynamics, etc. 

By the facts that the constitutive laws should be objective (coordinates-free) 
and physical variables appear always in one-to-one pairs (i.e. the Hill work-conjugacy 
principle in continuum mechanics [^), it is reasonable to assume that for a given 
natural system, there exists a certain objective measure e = A{x) : Xa C X ^ 
£a C iS and a stored energy W : £a ^ ^ such that the constitutive dual- 
ity relation e* = VT^(e) : iSa — > iPa C is canonical (i.e., one-to-one on 
£a X £a)- Such a system is the so-called canonical system and is denoted as 
§a = {{Xa,X*),{£a;£*);A,C} (see Chapter 4, [9]), where C = VW : £a ^ £* 
represents the constitutive mapping, (*, *) and *) denote the bilinear forms on 
X X X* and £ x £* , respectively. The system is called geometrically nonlinear 
(resp. linear) if the geometrical operator A is nonlinear (resp. linear); the system 
is called physically (or constitutively) nonlinear (resp. linear) if the constitutive 
operator C is nonlinear (resp. linear); the systems is called fully nonlinear (resp. 
linear) if it is both geometrically and physically nonlinear (resp. linear). 

The most simple geometrically linear system is controlled by the quadratic 
function n{x) = \{x, Ax) — (x, /), where A £ R"^" is a symmetrical matri^lfl. If 
A is positive (semi) definite, by Cholesky decomposition we know that there exists 
a matrix D : R" ^ R'" such that A = D'^D. Therefore, we have ^ {x, Ax) = 
\{Dx;Dx) = T{Dx) and T(y) is an objective function of y = Dx G R"^ . By 
the fact that any symmetrical matrix can be written in difference of two positive 
definite matrices, it turns out that any given quadratic function can be written in 
the so-called d.c. (difference of convex functions) form. 



6.2 Geometrically linear systems and Lagrangian duality 

In fact, the most popular Lagrangian in its original form is actually defined by 

n{x) = T{Dx) - U{x), 
^ The skew symmetric matrix As = |(A — A^) does not store energy since AsX = 0. 
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where the objective function T(y) : 3^a C M"* — > R is a kinetic energy, while 
U : Xa C — ^ R is a potential energy of the systerrQi which could be linear or 
convex such that n{x) is well-defined on the so-called kinetically admissible space 
= {x £ Xa \ Dx e [9_. For Newtonian mechanics, T{y) is quadratic and the 
objectivity of this kinetic energy ensures the validity of Newton's laws under the 
Galilean transformation; while for Einstein's special relativity theory, the objective 
function T(y) is strictly convex (see Chapter 2, [9|), which is an invariant under the 
Lorentz transformation. In either case, the so-called complementary energy T* (y*) 
can be uniquely defined on y* C by the classical Legendre transformation 
T*{y*) = sta{(y;y*) — T{y)\ y G 3^a} such that the original Lagrangian n{x) is 
equivalent to its mixed form 

L{x, y*) = {Dx- y*)-T* (y*) - U{x) -.X^^yl^W (52) 

which is the standard form in mathematical optimization. For conservative sys- 
tems, the Lagrangian should be a constants, therefore, the criticality condition 
VL(aj,y*) = leads to the well-known Euler-Lagrange equations: 

D*y* =VU{x), Dx = VT*{y*), (53) 

where D* is an adjoint of the linear operator D defined by {Dx; y*) = {x, D*y*) . 
By the canonical duality y = VT*{y*) <^ y* = VT{y), we have the equilibrium 
equation 

D*VT{Dx) = VU{x). 

Particularly, if U{x) = {x, f) is linear and T{y) = \ {y; Cy) is quadratic, where C 
is a linear operator, the equilibrium equation takes a particular symmetrical form 
D*CDx = f which is repeated throughout the field equations of mathematical 
physics [37] . 

For geometrically linear static systems, both the input and the configuration 
variables are time independent. In this case, the convex objective function T(y) is 
the so-called internal (or stored) energy and U(x) is the external potential, which 
should be linear U(x) = {x, f) such that its derivative Vf7(a;) = / is a given 
source of the system. Therefore, the Lagrangian form n{x) represents the total 
potential of the system, which is convex on Xk and its mixed form L{x,y*) is a 
saddle function on Xa x ya - Therefore, the traditional saddle Lagrangian duality 
theory links the convex primal problem min{77(a;)| x £ Xk} to a unique dual 
problem 

max{77*(y*) = -T*{y*)\ y* e K} , (54) 

where y* = {y* G D*y* = / G X* C R"} is the so-called statically admissible 
space. The objectivity of this dual problem is guaranteed by the objectivity of T{y) . 
By introducing a Lagrange multiplier x, which must be a solution to the primal 
problem (see Lagrange multiplier's law in Section 1.5 [9^), to relax the equilibrium 
constraint D*y* = / in y* , the Lagrangian is exactly the mixed form L{x,y*) 
and the one-to-one Lagrangian saddle min-max duality 

min n{x) = min max L{x,y*) = max min L{x,y*) = max n*{y*) 

^ The Lagrangian form was first introduced by W. Hamilton in classical mechanics and 
denoted by L = T — U , which is the standard notation extensively used from dynamical 
systems to quantum field theory (see 1291 ). 
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is called the mono- duality in canonical systems theory [9]- In mathematical eco- 
nomics, where the objective function T(Dx) is corresponding to the revenue, de- 
noted by R{x), and the potential U{x) is the cost function, denoted by C{x), 
then n{x) = R{x) — C{x) is the so-called total profit. For geometrically linear 
static problems, the cost function C{x) is usually linear, while the revenue R{x) 
is a concave objective function of certain measure (norm) of x in order to have 
maximum total profit n{x). 

In geometrically linear dynamical systems, the convex function T(y) is the 
kinetic energy and U (x) represents the total potential of the system. In this case, 
the Lagrangian form n{x) = T{Dx) — U{x) is the so-called total action, which is 
a d.c. (difference of convex) function. Since the mixed Lagrangian form L{x,y*) 
is no longer a saddle function, the well-known Hamiltonian 

H{x,y*) = {Dx-y*) - L{x,y*) = T*{y*) + U{x) 

was introduced, which is convex and has been extensively used in dynamical sys- 
tems. The Euler-Lagrange equations (|53|) is equivalent to the well-known canonical 
Hamiltonian equations 

Dx = Vy^H{x, y*), D*y* = V^H{x, y*). (55) 

Actually, although the Lagrangian is not a saddle function in convex Hamiltonian 
systems, it is a so-called super- critical function [9], and if the total potential U{x) 
is strictly convex on Xa C such that its Legendre conjugate U*{x*) can be 
uniquely defined on X* , then the canonical dual action of n{x) can still be defined 
by 

77* (y*) = max{L(a;,y*)| x e Xa} = U*{D*y*) - T*(y*) 

on 3^* = {y* £ y*\ D*y* € X*}, which is also a d.c. function. Therefore, instead 
of mono-duality in static systems, the convex Hamiltonian system is controlled by 
the so-called bi-duality theory. 

Theorem 5 (Bi-Duality Theorem) If {x,y*) is a critical point of the La- 
grangian L{x,y*), then x is a critical point of 77(a;), y* is a critical point of 
n*{y*) and n(x) = L{x,y*) = n*{y*). Moreover, if n = m, we have either 

n{x) = max77(a;) <^ max 77* (y*) = 77* (y*) (56) 

or 

n{x) = min n{x) <^ min 77* (y* ) = 77* (y*). (57) 

This bi-duality is actually a special case of the triality theory in geometrically 
linear systems, which was originally presented in Chapter 2 [9 for one-dimensional 
dynamical systems with a simple proof. This bi-duality reveals a stable periodical 
property in convex Hamiltonian systems. 
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6.3 Geometrically nonlinear systems and canonical duality 

Problems in geometrically nonlinear systems are usually nonconvex. Due to the 
fact that the geometrically linear operator D : — ^ can not change the 
convexity of the objective function, if W{x) is nonconvex and W{x) = T{Dx), 
the function T{y) is still nonconvex and its Legendre conjugate T*{y*) can not 
be uniquely defined p35| . It turns out that traditional Lagrangian duality theory 
can not be applied directly in this case. Although the Fenchel conjugate T'^{y*) = 
sup{(y;y*) — T{y)\ y G 3^a} can be uniquely defined, the function 

L{x,y*) = {Dx;y*) -THy*)-U{x) (58) 

is not the traditional Lagrangian form and the associate saddle min-max duality 
theory will produce the so-called duality gap in nonconvex optimization. 

Actually, in terms of U{x) = {x, f) — ^{x, Ax), the total complementary func- 
tion S{x, <;) defined by Q can be written as 

Six,^) = {A{x);^)-V*i<;)-Uix). (59) 

Comparing this S{x, <;) with either h{x, y*) or the mixed Lagrangian form L{x, y*) 
we can see that the fundamental difference between the canonical duality the- 
ory and other methods is the canonical transformation W{x) = V{A{x)) instead 
of the linear transformation W{x) = T{Dx) used in many other duality the- 
ories, including the Fenchel-Moreau-Rockafellar duality. In real applications, if 
the quadratic function U{x) is nonconvex, the mixed Lagrangian form L{x,y*) 
is nonconvex in x since D is linear. However, the total complementary function 
£"(*,<;) : X C — > R is always convex for <; G Sa and concave for ^ G Sa due 
to the geometrically nonlinear operator A{x) and its canonical dual variable <;. 
Therefore, S{x, <;) was also called the nonlinear Lagrangian in [9] and the extended 
Lagrangian in [11'. If the geometrical operator A{x) is quadratic and objective, 
the so-called yl-transformation [llj 

C/^(<;) = sta{(yl(a;);<;) - C/(a;)| x e X} (60) 

is actually the pure complementary gap function which is obtained from the com- 
plementary gap function Gap{x, <;) = ^{x, G{<;)x) by using the analytical solution 
form X = IG{<;)]-^F{<;). 

The geometrical nonlinearity in continuum physics means large deformation 
(far from equilibrium states) , which usually leads to bifurcation in static systems 
[34] and chaos in dynamical systems [12] . Therefore, geometrically nonlinear sys- 
tems are usually nonconvex. This is the reason why the geometrical nonlinearity 
was emphasized in the title of Gao and Strang's original work [21], although the 
system they studied is fully nonlinear and governed by a nonconvex/nonsmooth 
total (super) potential functional 

n{u)=W{A{u)) + F{u), (61) 

where W{e) is called the stored energy, which is a canonical function(al) such that 
the constitutive law e* = dW{e) is invertible on its effective domain; while F{u) 
is an external energy, which must be linear on the statically admissible space such 
that its Gateaux derivative dF{u) = —u* leads to the external force (source) field 
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(under the sign convention). The geometrically nonlinear operator e = A{u) in 
Gao and Strang's work should be an objective measure in order to satisfy certain 
well-known deformation laws (see Chapter 6, [9 ). Therefore, the complementary 
gap function Gap{u, e*) was naturally introduced. This objective function lays a 
foundation for the triality theory. 

Oppositely, in a recent paper entitled "Some remarks concerning Gao-Strang's 
complementary gap function" by Voisei and Zalinescu [30] , they choose quadratic 
functions as the external energy F{u) (see Examples 2, 4 and 5 in [40 ), and piece- 
wise linear function (see Example 1 in [30]) as the stored energy, they concluded: 
"About the (complementary) gap function one can conclude that it is useless at 
least in the current context" . Clearly, the piecewise linear function is not objec- 
tive and cannot store energy; while for those quadratic functions F(u) they listed, 
the dual variable u* = dF{u) depend on the configuration u. Such force field is 
called follower force. In this case, the system is not conservative and traditional 
variational methods do not apply. Unfortunately, similar counter-examples and 
conclusions are repeatedly presented in many other papers (see [38] and references 
cited therein) . 

Actually, in order to study nonconvex variational problems in dissipative sys- 
tems subjected to follower force field, a so-called rate variational method and the 
associated dual extremum principle were proposed in 1990 [17| . Also, Gao and 
Strang's work has been extended to general nonconvex dynamical systems to al- 
low F{u) as a quadratic function, but notations were changed (see [111112] '). In 
fact, if we let A{x) = {A{x), ^{x,Ax)} and W{A{x)) = V{A{x)) + ^{x,Ax), the 
general nonconvex problem {V) studied in this paper is simply a finite dimensional 
version of the Gao and Strang's general work in large deformation theory. This 
method has been repeatedly used in many Gao's papers (see [161143] 1. Particularly, 
if A{u) is a Cauchy-Riemann strain measure, then 

S{u, e*) = {A{uy,e*) - W*{e*) + F{u) (62) 

is the well-known Hellinger-Reissner complementary energy in finite deformation 
theorjjf]. Furthermore, if the complementary energy W* {e*) is replaced by (e; e*) — 
W{e), the total complementary energy S{u,e*) can be written in the so-called 
pseudo-Lagrangian (it was denoted as Lp{u,e* ,e) in [21] ) 

Shy,{u,e*,e) = W{e) + {A{u) - e-e*)+F{u), (63) 

and we have 

S{u, e*) = st&{Shw{u, e* ,e)\ e G £a}- 

In large deformation mechanics, ^^^{u^e* ,e) is called the Hu-Washizu gener- 
alized potential energy, proposed independently by Hai-Chang Hu in 1954 and 

* The Hellinger-Reissner energy was first proposed by Hellinger in 1914. After the external 
energy -F(it) and the boundary conditions in the statically admissible space = {« S Wa|e = 
A{u) g So} were fixed by Reissner in 1953, the associated variational statement has been 
known as the Hellinger-Reissner principle. However, the extremality condition of this principle 
was an open problem, and also the existence of pure complementary variational principles has 
been a well-known debate existing for over several decades in large deformation mechanics (see 
I30I V This open problem was partially solved by Gao and Strang's work and completely solve 
by the triality theory. While the pure complementary energy principle was formulated by Gao 
in 1999 [8]. 
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K. Washizu in 1955. The associated variational statement is the weh-known Hu- 
Washizu principle, which has important apphcations in computational mechanics 
of thin-walled structures, where the geometrical equation e = A{u) is usually pro- 
posed by certain geometrical hypothesis |15ll20] . 

It has been emphasized in many papers that the key step in the canonical dual- 
ity theory is to choose a geometrically reasonable measure ^ = A{x). It was shown 
in \25' that for a given nonconvex variational problem, the choice of A{x) may not 
be unique and different geometrically admissible operators could lead to different 
canonical dual problems. But all these canonical dual problems must be equivalent 
in the sense that they have the same set of solutions. Also for complex systems, two 
type of sequential canonical transformations were proposed (see Chapter 4, [9j). By 
the fact that the objectivity and canonical duality are fundamental to all natural 
systems, for any given real problem, as long as the geometrical operator A{x) can 
be chosen correctly such that the nonconvex objective function(al) can be recast 
by adopting a canonical form W{x) = V{A{x)), the canonical duality theory can 
be used to establish elegant theoretical results and to develop efficient algorithms 
for robust computations. The triality theory reveals an intrinsic duality pattern 
in nonconvex systems and should play important roles not only for solving a large 
class of challenging problems in nonconvex analysis and global optimization, but 
also for understanding, modeling, and simulation of complex systems. 

7 Conclusion Remarks 

Motivated by an open problem on the double-min duality in the triality theory 
that was left unaddressed since 2003, we have presented a mathematically rigorous 
proof for this theory based on the elementary linear algebra. Our results show that 
the triality theory holds strongly in the tri-duality form if the primal and its dual 
problems have the same dimension. Otherwise, both the canonical min-max and 
double-max duality statements hold strongly, but double-min duality statement 
holds weakly in a super-symmetric form. Additionally, a weak saddle-duality the- 
ory is proposed, which shows that when the complementary gap function Gap{x, <;) 
is negative, either the primal problem (V) (only if m < ti) or its canonical dual 
{V^) (only if m > 7i) could have saddle critical solutions. Therefore, this seven 
years old open problem is now solved completely and the triality theory is pre- 
sented in an elegant form as expected. 

The method adopted in this paper can be generalized for more general con- 
strained global optimization problems. As it is mentioned in Remark 3 that the 
primal problem {V) could be NP-hard if its canonical dual has no critical point 
in Sa ■ Also, the extremality conditions for those critical points {x, ^) are still un- 
known if the Hessian matrix G(^) of the gap function is indefinite. Although a 
general theorem on the existence and uniqueness of the canonical dual solution in 
Sa was proposed in [Tl^ , and some perturbation methods were discussed in [18] , 
detailed quantitative study on these topics is fundamentally important and critical 
for understanding and solving NP-hard problems. 
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